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ABSTRACT 



Twenty-one schools from the United States and Singapore 
participated in this evaluation of a high-school biology multimedia CD-ROM 
called BioBLAST [R] . Results include a summary of teacher descriptions of 
their curriculum-based justifications for using this software as well as a 
compilation of program features that teachers describe as facilitating 
successful implementation of new learning technologies. Analysis of teacher 
reports provides a profile of three types of strategies teachers used to 
implement the program. Analysis of student performance on pre/posttest shows 
how implementation strategies were associated with student gains in 
particular schools and classes. A comparison of student pre/posttest scores 
one year later with previous pre/posttest results suggests that changes in 
implementation strategies, teacher incentives, or a decrease in support 
services had a direct impact on student performance. Students with a wide 
range of prior knowledge were able to use the software and show performance 
gains appropriate to their knowledge level and experience. (Contains 10 
references . ) (Author/AEF) 
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Abstract 

Twenty-one schools from the United States and Singapore participated in this evaluation of a 
high-school biology CD-ROM called BioBLAST®. Results include a summary of teacher 
descriptions of their curriculum-based justifications for using this software as well as a 
compilation of program features that teachers describe as facilitating successful implementation 
of new learning technologies. Analysis of teacher reports provides a profile of three types of 
strategies teachers used to implement the program. Analysis of student performance on 
pre/posttests shows how implementation strategies were associated with student gains in 
particular schools and classes. A comparison of student pre/posttest scores one year later with 
previous pre/posttest results suggests that changes in implementation strategies, teacher 
incentives, or a decrease in support services had a direct impact on student performance. Students 
with a wide range of prior knowledge were able to use the software and show performance gains 
appropriate to their knowledge level and experience. 



Introduction 

This report examines technology implementation decisions made by teachers in the context of a 
particular multimedia high school biology software program called BioBLAST®, a computer-based, 
multimedia, learning environment for high school biology classes. This study examines the relationship 
between teacher implementation strategy and student performance. The software program described in 
this investigation was designed to help teachers address both NSES goals. Based on these goals, a 
scenario was created for a problem-based learning environment that put students in the role of NASA 
Advanced Life Support researchers. Student teams are challenged to use the software tools and resources 
to accomplish their goal: Design and test a plant-based life support system that can support a crew of six 
for at least three years without re-supply. 

Reports from national commissions, disciplinary groups, researchers, employers, faculty, and students 
repeat the call for instructional innovations in science, mathematics, engineering, and technology 
(SMET) education 1 . Across all these reports Springer, Stanne, and Donovan (1999, p. 22) find a 
consistent recommendation for a shift in emphasis from teaching to learning and greater use of, “active, 
collaborative, small-group work inside and outside the classroom.” The National Science Education 
Standards (NSES, 1996) recommends that teachers move away from methods of direct instruction to an 
inquiry-based approach to teaching. The report further recommends that teachers use alternative 
approaches to improving student learning that address both the method of instruction and the need to 
identify the content being addressed. Instruction should be student-centered rather than teacher-directed, 
and classroom curricula should be linked to real scientific issues and events (NSES, 1996). 

An evaluation study by Cook, Habib, Phillips, Settersten, Shagle, and Degirmencioglu (1999) reports that 
gains in student achievement were associated with schools that have a focused academic articulation of 
goals for student achievement. This finding is consistent with research by Phillips (1997) and Lipsitz, 
Mizell, Jackson, and Austin (1997) who report that student performance improvements are most closely 
related to academic focus and less tied to social climate issues. Cook, et al., (1999) acknowledge that the 
results of their evaluation study are limited by the fact that program implementation fell short at nearly 



1 Association for the Advancement of Science, 1989, 1990; Boyer Commission, 1998; National Research Council, 
1995, 1996; National Science Foundation, 1996; Peak, 1996. 
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all of the schools included in the study. They suggest that ways to improve and understand program 

implementation are greatly needed in order to understand student change. 

This study examined teacher implementation strategies by adapting the issues addressed by Cook et al. 

(1999) and Kerr (1999) to the context of high school science teachers involved as lead teachers in the 

field testing of the BioBLAST software. The following research questions are examined in this study. 

1. What were the academic and pedagogical goals that teachers described as their curriculum 
justification for using this software? 

2. What strategies for implementing the software were reported by teachers? 

3. How were implementation strategies expressed in student performance as evidenced in the 
pre/posttest scores? Are these student performance measures consistent with teacher reports? 

4. What does the comparison of teacher implementation strategies and student performance during the 
different periods of the program cycle suggest regarding classroom application of this curriculum 
supplement? 



Method 

BioBLAST was developed by the NASA Classroom of the Future in collaboration with the National 
Aeronautics and Space Administration (NASA) Advanced Life Support research program. The virtual 
reality interface of the CD-ROM draws students into a futuristic, problem-solving scenario that is situated 
in a simulated, lunar research facility. In the virtual lunar base environment, students use graphical 
simulation tools and resources to prepare for their mission goal: to design and test a model for a plant- 
based, life-support system that can sustain a crew of six for three years. Along with the goal of helping 
teachers move to an inquiry-based approach to teaching, a secondary goal of this program is for students 
to have access to current life science research in the process of investigating a legitimate problem. 

The data collection techniques include teacher surveys, observations of teachers and students in their 
classroom environment, interviews with teachers, pre/posttests, and student and teacher reports. Twenty- 
one high schools from urban, suburban, and rural areas of the United States and Singapore participated in 
the classroom testing of this software. The school applications were self-selected in all but two cases. 

The specific schools included were selected from the group of applications based on geographic 
distribution, access to technology required to run software, and level of interest in learning alternative 
teaching techniques as expressed by teachers in their personal essays. 

This study compares teacher implementation strategies with student performance during the final stage of 
the formative evaluation cycle (beta testing) with student performance during the pre-publication cycle 
when professional development support for teachers was no longer available. Participants differed in the 
location and level of technology available at their school and in the level of administrative support for 
implementation of curriculum-related instructional technology. This analysis of student outcomes based 
on comparison of pre/posttest scores involves a subset of the larger group who fully implemented the 
program and returned the completed pre/posttest materials. Table 1 provides a summary description of 
schools involved in the larger and subset groups. 



Results 

The following questions were examined in analysis of the data collected from teachers reports, surveys 
and interviews; student reports, pre/posttests, and surveys; and observation of on site and off site 
implementations of the program. 

1. What were the academic and pedagogical goals that teachers described as their curriculum 
justification for using this software? 

Many teachers described their use of BioBLAST software as offering an opportunity to change their 
methods of teaching to incorporate use of technology within the context of instruction and to include 



2 BioBLAST was produced and developed by NASA Classroom of the Future™ for the National Aeronautics and 
Space Administration through Cooperative Agreement No. NCC5-203. awarded to the NASA Classroom of the 
Future, Center for Educational Technologies, Wheeling Jesuit University, Wheeling, WV 26003. 
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more cooperative learning group activities. Moving to group work was cited by one third of the teachers 
as a greater challenge to their teaching practice than incorporating the new technology. Teachers also 
reported that Internet access and at least class email accounts accessible within the classroom were an 
important technology component of this program. These communication resources allowed students to 
ask questions to NASA experts and find current resources on the Web. Teachers commented that 
students were drawn into the QuickTime virtual reality interfaces since many video games that are 
popular among teenagers use this kind of interface. The “BaBS” culminating simulation was captivating 
to nearly all students and was the students most favorite aspect of the BioBLAST program. Students also 
responded favorably to the scenario link with current NASA research and laboratory exercises that had 
real connections with NASA experiments. 

A summary of program features that teachers reported as most important to them in selecting new 
resources to adopt and implement is provided in Table 2. An important feature that emerges from this 
summary is that teachers want software/technology tools that will support their curriculum goals and help 
them grow. Teachers report changes in their teaching position, restructuring of existing course, the 
creation of new courses, and the implementation of alternative scheduling at their school based on 
experiences with the BioBLAST program as a “test-case” technology-integration program. 



Table 1. Summary of Schools Involved in the Larger and Subset Evaluation Groups 



School 


Student 

Total 


Asian 


Black 


Hispanic 


Caucasia 

n 


Native 

America 

n 


Pacific 

Islander 


School 

Affiliation 


School 

location 


6.12.4.6 


280 


11 


160 


36 


71 


2 


0 


public 


urban 


8.9.10.16 


14 


6 


0 


0 


4 


0 


3 


public 


rural 


8. 9. 5. 7 


35 


29 


0 


1 


l 


0 


3 


public 


urban 


13.9.X 


60 


0 


56 


1 


2 


1 


0 


public 


urban 


13.14.6.8 


101 


8 


1 


1 


90 


0 


0 


public 


urban 


14.25.13.17 


60 


2 


6 


0 


51 


0 


0 


public 


suburban 


14.4.13.19 


25 


0 


0 


0 


20 


5 


0 


public 


rural 


14.10.3.4 


130 


7 


1 


4 


118 


0 


0 


private 


suburban 


14.13.X 


25 


0 


0 


0 


0 


25 


0 


public 


rural 


14.25.9.9 


19 


0 


0 


18 


0 


0 


0 


public 


rural 


14.25.7.11 


147 


16 


7 


7 


110 


0 


6 


private 


urban 


13.15.16.22 


150 


1 


13 


46 


87 


0 


30 


public 


suburban 


15.8.1.1 


118 


2 


2 


0 


114 


0 


0 


public 


suburban 


15.8.8.12 


48 


0 


1 


0 


47 


0 


0 


public 


suburban 


16.1.12.18 


85 


4 


9 


4 


68 


0 


0 


public 


suburban 


3.1.0.X 


36 


0 


0 


29 


6 


0 


0 


public 


urban 


22.1.2.3 


119 


6 


36 


24 


54 


0 


0 


public 


suburban 


21.22.11.17 


36 


0 


0 


0 


35 


0 


0 


public 


rural 


23.22.X 


25 


0 


1 


0 


25 


0 


0 


private 


suburban 


Raffles 


42 


22 


0 


0 


0 


0 


20 


private 


urban 


College Prep 


20 


1 


0 


0 


16 


1 


2 


private 


suburban 


TOTAL 


1575 


115 


293 


171 


919 


34 


64 


private - 5 
public - 16 


urban - 7 
suburban - 9 
rural - 5 



Table 2. Program Features that Teachers Report Facilitates Successful Implementation: 



Has administrative and department- level support; 


Designed to help teachers grow 


Links to real-world event(s); 


Includes a description of prior knowledge required; 


Includes a vision of how to makes learning more 
meaningful; * 


Includes a description of computer and laboratory 
access time required; 


Includes guidelines for enhancing classroom learning 
that demonstrates effective use of technology through 
exposure to new methods, materials, and other 


Includes a taxonomy of content and process skills 
addressed so that teachers can link program to school 
framework for basic math and science objectives; 
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teachers; 




Activities encourage student problem solving and 
facilitate cooperative group work 


Addresses teacher needs for appropriate assessment 
material 




Applicable in small units or chunks and still effective 



2. What strategies for implementing the program were reported by teachers? 

Teachers used the laboratory and computer-based investigations provided by the software development 
team and incorporated these materials into their academic grading system. Teachers adapted the 
suggested sequence outlined by the project development team to suit the needs of their students, their 
school curriculum guidelines, and the time they had available in their course schedule. Teachers selected 
implementation strategies based on topics and scientific inquiry process skills that they believed their 
students most needed to address. The software development group suggested that teachers view their 
implementation of the software as having four phases much like the four phases of the problem based 
learning approach to teaching. The four phases of BioBLAST were presented to teachers as: Phase 1, 
Orientation to the problem; Phase 2, Research and analysis of the sub-units of the problem; Phase 3, the 
Mission in which the sub-units are integrated and combined for testing possible solutions to the problem; 
and Phase 4, the Report in which students present a summary of data collected, data analysis, results of 
initial testing, and recommendations for solutions and continued research. 

Based on analysis of teacher reports, surveys, documentation of email communications, and observation 
of teachers in their classroom, we found that teachers implemented the software in three distinct ways. 
The “Lab” group focused most of their time on skill-building activities and one or more of the laboratory 
activities. This group allowed their students to interact with introductory materials and structured 
exercises embedded within the software simulation programs. Teachers using this implementation style 
reported that their students did not have science content and process skills required for the more advanced 
activities. The “Simulator” group completed the Orientation, Research, and Mission phases of the 
program. This group allocated at least three class periods (45 minutes each) for their students to 
complete the "BaBS" model building simulation in which previous plant, human, and resource recycling 
research is compiled and integrated and tested. The format and assessment of the Report phase were more 
open-ended than previous activities, and this may be why some teachers did not implement this phase. 
The group called the “Researcher” group completed all four phases of the project and had their students 
complete a project report as their culminating activity for the program. 

3. How were implementation strategies expressed in student performance as evidenced in the 
pre/posttest scores? Are these student performance measures consistent with teacher reports? 

Not all teachers submitted pre/posttest results. Teacher surveys and reports were required, but 
pre/posttests were not. A small percentage of the Lab group submitted their pre/posttest results, and 
teachers reported that they did not feel that they gave their students the full benefit of the program 
because they did not use the simulations. The entire Simulator group and all but one of the Researcher 
group submitted their 

students’ pre/posttests for evaluation. 

The pre/posttest consisted of 16 questions, seven of which were multiple choice, the others requiring 
short answer and one concept-mapping responses. Analysis of pre/posttest scores on the seven multiple- 
choice questions indicates that student performance was significantly improved in all but two cases. The 
paired T-test analysis shows huge gains in nine of the eleven cases. Table 2 provides a summary of the 
main effect of the T-test score for each school. In one of the cases where student scores overall did not 
improve, the student pretest scores were already nearly perfect on the multiple choice test questions. In 
the other case, student scores improved, but this improvement was only significant on one question, and 
not all seven or on overall score. Teacher reports from the eleven participating schools expressed 
confidence that the program helped them successfully implement new teaching practices that 
incorporated student-centered and inquiry-based approaches to learning. Below are a few excerpts of 
teacher observations about student learning from the software program. 

• “BioBLAST made student learning more meaningful; students are beginning to show more 

proficiency in their use of technology; the simulations encourage student thinking because they have 
to ask themselves why did they only survive that long, what happened?” 
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• “Later in the [biology] course, students would reference what they saw in BioBLAST as a reference 
for other bio curriculum activities; BioBLAST seemed to provide a good transfer, a good bridge, and 
a solid base for students to refer back to...” 

• “Students were frustrated by the technology, but were also engaged by it... [through the use of this 
program] I accomplished the goal of incorporating more math, graphing, and research skill-building 
into student work; working in teams was most challenging, but in some cases most rewarding for 
students.” 

A doctoral research study (Cain, 2000) was conducted during the alpha testing of the software that 
analyzed student group discussions while using the Mission Phase simulation. One of the questions 
examined in this research study was “In what ways did the content knowledge change after the initial 
phase of the simulation-based learning activity?” Based on his analysis of the data, Cain reported 
improvements in student content knowledge in both early and later trials with the simulation. Six out of 
the sixteen content areas in which Cain noted deficiencies in early trials improved with continued use of 
the simulator. Improvements in content knowledge, similar to those noted by Cain (2000), was also 
evidenced in student pre/posttest scores in the following areas: [Question 2] how plants obtain nitrogen; 
[Question 5] understanding of the interdependence of plants and animals; [Question 8] graph 
interpretation of physical science concept; [Question 9] experimental design for plant science 
investigation; and [Question 12] in-depth understanding of human and plant respiration and 
photosynthesis. 



Table 3. Summary of Effect Sizes for Paired T-test Scores* 



Teacher Code 


Effect Size for Paired T-test by Teacher - Beta 
Group 


Effect Size for Paired T-test by Teacher - Pre- 
Release Group 


8.9.10.16 


1.3 




8.9.5 .7 


0.91 




13.14.6.8 


0.31 




14.10.3.4 


0.4 




14.25.9.9 


1.13 




14.25.7.11 


0.72 


0.43 


15.8.1.1 


0.91 


0.64 


15.8.1.2 


1.24 




15.8.8.12 


1.16 




21.22.14.2 


1.06 




19.1.0.X 


0.08 




Average 


0.84 





* Values of .4 and higher indicate a main effect. Bold numbers indicate significant effect size. 



4. What does the comparison of teacher implementation strategies and student performance during the 
different periods of the program cycle suggest regarding classroom application of this curriculum 
supplement? 

As Table 3 shows, teacher implementation during the Beta testing process was more successful than their 
implementation in the later year when there was minimal support and no in-service workshop provided. 
During the Beta testing, teachers received a small stipend to cover the cost of their reporting, 
administering, and handling of evaluation materials. Teachers also attended a 5-day off-site workshop 
the summer before the Beta testing in the classroom began. Weekly updates regarding the project were 
emailed to teachers during the beta testing, and teachers were encouraged to contact project team 
members if they had questions or needed assistance. 

As reported in the research study by Cain (2000) and supported by analysis of teacher surveys and reports 
student learning improved and continued through continued use of the simulators. Later implementations 
of the program may not spend the same amount of time preparing the students for effective use of the 
simulators and may not allow time for continued trials, analysis, and comparisons of simulator trials. In 
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addition to the professional development and incentive programs, giving students two to four separate 
sessions to use the simulators, analyze results, make modifications, and compare outcomes is a necessary 
interactive process for effective use of the simulations. More research is needed to identify what support 
or feature associated with the beta testing is most critical to facilitating the most successful, long-term 
implementation success. Since the number of pre- and posttests available for analysis were greatly 
reduced, it is difficult to conclude whether the trend found in the two cases documented was consistent 
across all implementations. 



Discussion 

What generalizations can be made regarding ways to improve program implementation? Can effective 
implementation strategies improve student achievement? Student performance improvements were 
statistically significant in all but two cases. The indication of reduced (although still demonstrating 
statistically significant improvement) student achievement in the pre-release (post-beta) test group 
reflects the time-sensitive nature of program implementation. The list of program features that teachers 
report facilitates successful software implementation (see Table 2) suggests that if any one of these 
eleven factors changes, the implementation strategy may be curtailed or disrupted. Teacher reports and 
survey data indicates that teachers want a clear set of recommendations for an activity sequence, time to 
allocate for each activity, and suggestions for effective use of the simulators with student groups to 
support their implementation strategies. This information was not available during formative evaluation, 
but can be made available on the Web site to new teachers using the program. 

Several interesting observations from this research suggest ways in which software and other learning 
environment technology tools can motivate and engage students who have previously shown little or no 
interest in a given course. Analysis of student performance from the pre/posttest scores showed that 
software implementation that included use of the simulators exhibited gains for groups that were weak in 
fundamental content as well as for groups that were ready for more advanced concepts. By allowing 
multiple ways to implement the software, students with a wide range of prior knowledge were able to use 
the software and show performance gains appropriate to their knowledge level and experience. 
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